feat(cli): `cdk rollback` #31407

rix0rrr · 2024-09-11T14:57:54Z

Add a CLI feature to roll a stuck change back.

This is mostly useful for deployments performed using --no-rollback: if a failure occurs, the stack gets stuck in an UPDATE_FAILED state from which there are 2 options:

Try again using a new template
Roll back to the last stable state

There used to be no way to perform the second operation using the CDK CLI, but there now is.

cdk rollback works in 2 situations:

A paused fail state; it will initiating a fresh rollback (on CREATE_FAILED, UPDATE_FAILED).
A paused rollback state; it will retry the rollback, optionally skipping some resources (on UPDATE_ROLLBACK_FAILED -- it seems there is no way to continue a rollback in ROLLBACK_FAILED state).

cdk rollback --orphan <logicalid> can be used to skip resource rollbacks that are causing problems.

cdk rollback --force will look up all failed resources and continue skipping them until the rollback has finished.

This change requires new bootstrap permissions, so the bootstrap stack is updated to add the following IAM permissions to the deploy-action role:

                  - cloudformation:RollbackStack
                  - cloudformation:ContinueUpdateRollback

These are necessary to call the 2 CloudFormation APIs that start and continue a rollback.

Relates to (but does not close yet) #30546.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Add a CLI feature to roll a stuck change back. This is mostly useful for deployments performed using `--no-rollback`: if a failure occurs, the stack gets stuck in an `UPDATE_FAILED` state from which there are 2 options: - Try again using a new template - Roll back to the last stable state There used to be no way to perform the second operation using the CDK CLI, but there now is. `cdk rollback` works in 2 situations: - A paused fail state; it will initiating a fresh rollback. - A paused rollback state; it will retry the rollback, optionally skipping some resources. `cdk rollback --force` will look up all failed resources and continue skipping them until the rollback has finished.

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

packages/@aws-cdk-testing/cli-integ/tests/cli-integ-tests/cli.integtest.ts

packages/aws-cdk/README.md

iliapolo · 2024-10-01T09:50:39Z

packages/aws-cdk/lib/cdk-toolkit.ts

+        print('\n✨  Rollback time: %ss\n', formatTime(elapsedRollbackTime));
+      } catch (e: any) {
+        error('\n ❌  %s failed: %s', chalk.bold(stack.displayName), e.message);
+        throw new Error('Rollback failed (use --force to orphan failing resources)');


Would be better to accumulate errors to avoid a poison pill.

What do you mean?

Oh so I think I was commenting from the position that if a stack cannot be rolled back, we should throw an error - in this case a single faulty stack can prevent rolling back others, so we need to accumulate errors.

If you insist on no-oping for a unrollable stack, this is fine, but I still think we should error out.

So I see that right now you do throw an error on ROLLBACK_FAILED state - doesn't this mean you need to swallow the error here and proceed? to avoid the poison pill?

packages/aws-cdk/lib/cli.ts

packages/aws-cdk/test/api/stack-activity-monitor.test.ts

packages/aws-cdk/test/util/stack-monitor.test.ts

iliapolo · 2024-10-02T09:06:33Z

packages/@aws-cdk-testing/cli-integ/resources/cdk-apps/rollback-test-app/app.js

+ * It contains resources r1 and r2, where r1 gets deployed first.
+ *
+ * - PHASE = 1: both resources deploy regularly.
+ * - PHASE = 2: r1 gets updated, r2 will fail to update, and r1 will fail its rollback.


Needs to be adjusted for phases 2a and 2b. I'm good with just removing this and let the tests speak for themselves.

packages/aws-cdk/lib/api/deployments.ts

iliapolo · 2024-10-02T09:18:34Z

packages/aws-cdk/lib/cdk-toolkit.ts

+        print('\n✨  Rollback time: %ss\n', formatTime(elapsedRollbackTime));
+      } catch (e: any) {
+        error('\n ❌  %s failed: %s', chalk.bold(stack.displayName), e.message);
+        throw new Error('Rollback failed (use --force to orphan failing resources)');


So I see that right now you do throw an error on ROLLBACK_FAILED state - doesn't this mean you need to swallow the error here and proceed? to avoid the poison pill?

aws-cdk-automation · 2024-10-02T09:21:49Z

➡️ PR build request submitted to test-main-pipeline ⬅️

A maintainer must now check the pipeline and add the pr-linter/cli-integ-tested label once the pipeline succeeds.

mergify · 2024-10-02T11:45:08Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2024-10-02T12:16:19Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 9fcab7c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2024-10-02T12:16:21Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

github-actions · 2024-10-02T12:16:43Z

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

rix0rrr requested a review from a team September 11, 2024 14:57

aws-cdk-automation requested a review from a team September 11, 2024 14:58

github-actions bot added the p2 label Sep 11, 2024

mergify bot added the contribution/core This is a PR that came from AWS. label Sep 11, 2024

rix0rrr had a problem deploying to test-pipeline September 11, 2024 14:58 — with GitHub Actions Failure

aws-cdk-automation previously requested changes Sep 11, 2024

View reviewed changes

aws-cdk-automation added the pr/needs-cli-test-run This PR needs CLI tests run against it. label Sep 11, 2024

rix0rrr added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Sep 11, 2024

Fix tests

c16c7a8

rix0rrr had a problem deploying to test-pipeline September 12, 2024 09:56 — with GitHub Actions Failure

rix0rrr added pr/do-not-merge This PR should not be merged at this time. pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested labels Sep 12, 2024

aws-cdk-automation added pr/needs-maintainer-review This PR needs a review from a Core Team Member and removed pr/needs-cli-test-run This PR needs CLI tests run against it. labels Sep 12, 2024

github-actions bot mentioned this pull request Sep 16, 2024

Weekly PR metrics report #31456

Closed

ericzbeard mentioned this pull request Sep 18, 2024

Implement continue and retry aws-cloudformation/rain#527

Open

Merge remote-tracking branch 'origin/main' into huijbers/cli-rollback

99bc2d9

rix0rrr had a problem deploying to test-pipeline September 25, 2024 09:53 — with GitHub Actions Failure

Unit tests

4245ff5

rix0rrr had a problem deploying to test-pipeline September 25, 2024 12:38 — with GitHub Actions Failure

Remove unused imports

be22319

rix0rrr had a problem deploying to test-pipeline September 25, 2024 12:54 — with GitHub Actions Failure

rix0rrr self-assigned this Sep 25, 2024

Fix fake class incompatibilities

c6d9b19

rix0rrr had a problem deploying to test-pipeline September 26, 2024 12:42 — with GitHub Actions Failure

iliapolo reviewed Oct 1, 2024

View reviewed changes

Initial set of review comments

bf7bef3

rix0rrr had a problem deploying to test-pipeline October 1, 2024 13:07 — with GitHub Actions Failure

Merge remote-tracking branch 'origin/main' into huijbers/cli-rollback

55f7800

rix0rrr had a problem deploying to test-pipeline October 1, 2024 13:08 — with GitHub Actions Failure

Add integ test without --force

75acb87

rix0rrr had a problem deploying to test-pipeline October 1, 2024 13:16 — with GitHub Actions Failure

Prevent infinite loop

453bb89

rix0rrr had a problem deploying to test-pipeline October 1, 2024 13:18 — with GitHub Actions Failure

Make ROLLBACK_FAILED an error

ae05b6f

rix0rrr had a problem deploying to test-pipeline October 1, 2024 14:36 — with GitHub Actions Failure

No error

3d3e97d

iliapolo reviewed Oct 2, 2024

View reviewed changes

rix0rrr temporarily deployed to test-pipeline October 2, 2024 09:19 — with GitHub Actions Inactive

iliapolo approved these changes Oct 2, 2024

View reviewed changes

Update docs

18c7ea6

rix0rrr had a problem deploying to test-pipeline October 2, 2024 09:40 — with GitHub Actions Failure

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Oct 2, 2024

rix0rrr removed the pr/do-not-merge This PR should not be merged at this time. label Oct 2, 2024

Merge branch 'main' into huijbers/cli-rollback

9fcab7c

mergify bot had a problem deploying to test-pipeline October 2, 2024 11:46 Failure

mergify bot merged commit 0755561 into main Oct 2, 2024
11 of 12 checks passed

mergify bot deleted the huijbers/cli-rollback branch October 2, 2024 12:16

github-actions bot locked as resolved and limited conversation to collaborators Oct 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): `cdk rollback` #31407

feat(cli): `cdk rollback` #31407

rix0rrr commented Sep 11, 2024 •

edited

Loading

aws-cdk-automation left a comment

iliapolo Oct 1, 2024

rix0rrr Oct 1, 2024

iliapolo Oct 1, 2024

iliapolo Oct 2, 2024

iliapolo Oct 2, 2024

iliapolo Oct 2, 2024

aws-cdk-automation commented Oct 2, 2024

mergify bot commented Oct 2, 2024

aws-cdk-automation commented Oct 2, 2024

mergify bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

feat(cli): cdk rollback #31407

feat(cli): cdk rollback #31407

Conversation

rix0rrr commented Sep 11, 2024 • edited Loading

aws-cdk-automation left a comment

Choose a reason for hiding this comment

iliapolo Oct 1, 2024

Choose a reason for hiding this comment

rix0rrr Oct 1, 2024

Choose a reason for hiding this comment

iliapolo Oct 1, 2024

Choose a reason for hiding this comment

iliapolo Oct 2, 2024

Choose a reason for hiding this comment

iliapolo Oct 2, 2024

Choose a reason for hiding this comment

iliapolo Oct 2, 2024

Choose a reason for hiding this comment

aws-cdk-automation commented Oct 2, 2024

mergify bot commented Oct 2, 2024

aws-cdk-automation commented Oct 2, 2024

AWS CodeBuild CI Report

mergify bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

feat(cli): `cdk rollback` #31407

feat(cli): `cdk rollback` #31407

rix0rrr commented Sep 11, 2024 •

edited

Loading